Automatic revisions to document clauses based on clause type

ABSTRACT

A document management system can include an artificial intelligence-based document manager that can perform one or more predictive operations based on characteristics of a user, a document, a user account, or historical document activity. For instance, the document management system can apply a machine-learning model to determine how long an expiring agreement document is likely to take to renegotiate and can prompt a user to begin the renegotiation process in advance. The document management system can detect a change to language in a particular clause type and can prompt a user to update other documents that include the clause type to include the change. The document management system can determine a type of a document being worked on and can identify one or more actions that a corresponding user may want to take using a machine-learning model trained on similar documents and similar users.

TECHNICAL FIELD

The disclosure generally relates to the field of document management,and specifically to predicting actions a user may want to take withregards to documents.

BACKGROUND

Online document management systems can be used to provide, create, andreview documents. Conventional document analysis environments mayprovide users with tools to view individual documents but lackfunctionality that makes sufficient use of metadata associated with thedocuments to improve user interactions with the documents. To keep acollection of documents up to date, a user must keep track of upcomingdeadlines associated with agreements in various documents, remember toupdate similar clause language across a portfolio of related documents,and manually perform additional actions related to editing and reviewingdocuments. To provide an improved, efficient, and more reliable documentinteractions experience to the user, there is a need for a system thatassists a user with document upkeep, analysis, and review.

SUMMARY

To help a user keep track of deadlines and expiration dates related todocuments, a document management system identifies upcoming expiringagreements and uses machine learning models to predict how long it willtake the user to renegotiate the agreement with a counterparty to theagreement. In this way, the document management system can prompt theuser to begin the renegotiation process at least a predicted interval oftime ahead of the expiration date. The system identifies an expirationdate associated with each document in a set of documents that areassociated with the user. The document management system can then applymachine learning models to each of the documents to determine, for thosedocuments with expiration dates, how long it is likely to take the userto re-negotiate the agreement with a counterparty. Working backward fromthe expiration date of the document and leaving at least the predictedamount of time for a re-negotiation, the document management systemnotifies the user of the upcoming expiration date of the agreement andof the predicted time for re-negotiation. The user can submit a requestfor the document management system to begin preparing the documents forthe re-negotiation process.

The document management system also helps users to keep clause languageconsistent across multiple documents. As a user edits language of onedocument, the document management system can provide a list of otherdocuments associated with the user that have the same or similar clauselanguage and may give the user the option to automatically update thelanguage to match the edits made to the first document. The documentmanagement system identifies that a user has made an edit to a clause ofa first document and queries a database to find a set of documentsassociated with the user that have the same clause in its or near tooriginal form. The document management system then updates a userinterface to notify the user of the related documents with similarclauses and the user can provide feedback via the interface to requestthat the similar clauses in one or more of the identified documents beupdated to match the edited clause of the first document.

The document management system also makes it easier for a user to accessand request actions with respect to individual documents. Machinelearning models are trained to identify a document type when given adocument or a set of document features. The document management systemapplies the models to a document that a user is editing or reviewing.Based on the document type predicted by the model and the documentcontents, the document management system identifies a set of actionsthat can be taken on the document. The set of actions is presented tothe user in a user interface and the user can easily request that one ormore of the actions be taken by the document management system to updatethe document automatically.

BRIEF DESCRIPTION OF DRAWINGS

The disclosed embodiments have other advantages and features which willbe more readily apparent from the detailed description, the appendedclaims, and the accompanying figures (or drawings). A brief introductionof the figures is below.

Figure (FIG. 1 is a high-level block diagram of a system environment fora document management system, in accordance with an example embodiment.

FIG. 2 is a high-level block diagram of a system architecture of adocument management system, in accordance with an example embodiment.

FIG. 3 is a flow diagram illustrating training and application processesfor a negotiation prediction model 325, in accordance with an exampleembodiment.

FIG. 4 illustrates an interface 400 of a document analytics dashboardfor reviewing upcoming expiration dates of agreement documents, inaccordance with an example embodiment.

FIG. 5 illustrates an example process for identifying upcoming expiringagreement documents and predicting a length of time for re-negotiatingthe agreement, in accordance with an embodiment.

FIG. 6 illustrates an interface of a document editing dashboard forreviewing similar clauses across separate documents, in accordance withan example embodiment.

FIG. 7 illustrates an example process for flagging clauses in documentsfor updating when a similar clause is updated in another document, inaccordance with an embodiment.

FIG. 8 is a flow diagram illustrating training and application processesfor a document type prediction model, in accordance with an exampleembodiment.

FIG. 9 illustrates an interface of a document editing dashboard forviewing and selecting suggested actions related to a document, inaccordance with an example embodiment.

FIG. 10 illustrates an example process for identifying actions that auser can request for a document, in accordance with an embodiment.

DETAILED DESCRIPTION

The Figures (FIGS.) and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. A letterafter a reference numeral, such as “120A,” indicates that the textrefers specifically to the element having that particular referencenumeral. A reference numeral in the text without a following letter,such as “120,” refers to any or all of the elements in the figuresbearing that reference numeral.

The figures depict embodiments of the disclosed system (or method) forpurposes of illustration only. One skilled in the art will readilyrecognize from the following description that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles described herein.

Overview

A document management system can assist users in identifying upcomingexpiring agreements. This can be helpful for example if a client hasmultiple agreements that will require renewal negotiations in the nearfuture. The document management system uses a trained negotiationprediction model to predict how long it is likely to take for agreementsto be re-negotiated. Based on the prediction, the document managementsystem may prompt a user of a client device to begin the renegotiationprocess at least a predicted interval of time ahead of the expirationdate of the agreement. Although examples herein describe agreementre-negotiations, this process may be used in other situations, such asfor documents with associated filing deadlines, or for documents whichare regularly generated and updated or that are time-sensitive, such asmonthly calendars, and bi-monthly supply request forms.

The document management system also monitors standard clauses includedby a user in documents and, when a clause is updated, flags otherdocuments that include the same clause for updating. In this way, a usercan be notified of other documents that the user may also want to updateto include the language of the updated clause. If the difference betweenthe instance of each clause in a flagged document and the updated clauselanguage exceeds a threshold value, the documents may be presented tothe user within an interface that enables the user to automate theprocess of updating the language of the outdated clause instances in thedocuments.

The document management system additionally uses machine learning modelsto identify a type of document with which a user is interacting and tosuggest actions that can be taken on the document. Examples of actionsthat may be taken with respect to a document include replacing text withfields, replacing clauses with pre-approved versions of the clauses, andsynchronizing the document with a third-party system. Action options maybe based on actions taken by similar users, actions done to similardocuments, and actions taken in similar contexts (e.g., at similar timesof the year). In one embodiment, the document management system presentsthe recommended actions within an interface that enables a user to viewand select desired actions from a list of suggested actions and thatenables the user to automate the performance of the actions if requested

System Environment

The system environment described herein can be implemented within anonline document system, a document execution system, or any type ofdigital transaction management platform. It should be noted thatalthough description may be limited in certain contexts to a particularenvironment, this is for the purposes of simplicity only, and inpractice the principles described herein can apply more broadly to thecontext of any digital transaction management platform. Examples caninclude but are not limited to online signature systems, online documentcreation and management systems, collaborative document and workspacesystems, online workflow management systems, multi-party communicationand interaction platforms, social networking systems, marketplace andfinancial transaction management systems, or any suitable digitaltransaction management platform.

FIG. 1 is a high-level block diagram of a system environment for adocument management system, in accordance with an example embodiment.The system environment enables client systems associated with a documentmanagement system 140 to create and manage digital documents, e.g., forelectronic execution to one or more receiving parties. A client mayreview, modify, and execute generated documents. As illustrated in FIG.1 , the system environment includes documents for analysis 110, one ormore client devices 120 (e.g., client device 120A and client device120B), a document corpus 130, and a document management system 140, eachcommunicatively interconnected via a network 180. In some embodiments,the system environment includes components other than those describedherein. For clarity, although two client devices 120 are shown in FIG. 1, alternate embodiments of the system environment can have any number ofclient devices 120. For the purposes of concision, the web servers, datacenters, and other components associated with an online systemenvironment are not shown in FIG. 1 .

The documents for analysis 110 are analyzed to identify upcomingexpiration dates of stored documents for renegotiation purposes, toidentify changes to clause language and portions of other documents withsimilar clause language that may need to be updated, and to identifydocument types for action recommendations based on actions taken withrespect to similar documents and/or similar users. Examples of documentsfor analysis 110 include but are not limited to: a sales contract, apermission slip, a rental or lease agreement, a liability waiver, afinancial document, an investment term sheet, a purchase order, anemployment agreement, a mortgage application, and so on. Each documentfor analysis 110 can include one or more clauses. Clauses are portionsof a document, and may include text, images, or other material of anylength. In some embodiments, a clause corresponds to a legal clause, abusiness clause, financial agreement text, and the like. A givendocument in the documents for analysis 110 may include multiple clausesthat each correspond to a different header within the document. Clausesmay further be associated with one or more clause types thatcharacterize content of the clause as corresponding to a particularcategory of the clause. Examples of clause types include but are notlimited to an indemnity clause, a merger and integration clause, aseverability clause, a fees clause, a damages clause, a pricing clause,a purchase clause, a payment clause, and so on. It should be noted thatnot every portion of a document or of text within a document may beconsidered a “clause” for the purposes of the description here.

A client device 120, such as client device 120A, provides the set ofdocuments for analysis 110 to the document management system 140 (orprovides instructions to create documents with information from theclient device 120). Although examples are given herein in the context ofa set of documents for analysis 110, the document management system 140can coordinate the creation, viewing, editing, analyzing, and signing ofany number of documents (e.g., thousands, millions, and more) for anynumber of users or accounts, and for any number of entities ororganizations. The client device 120 is a computing device capable oftransmitting or receiving data over the network 180. The client device120 may be a conventional computer (e.g., a laptop or a desktopcomputer), a server, a cell phone, or a similar device. The clientdevice 120 enables a user to create a set of documents for analysis 110and to provide the set of documents for analysis 110 to the documentmanagement system 140. After the document management system 140 analyzesor otherwise makes use of the set of documents for analysis 110, theclient device 120 may provide information to a user of the client device120 describing any results related to the analysis (e.g., via display ofa user interface). In some embodiments, the client device 120 receivesan interface for display from the document management system 140.

The document corpus 130 includes document information for the documentmanagement system 140. The document corpus 130 may include a set oftraining documents (e.g., for use by the document management system 140in training machine learning models). Documents in the document corpus130 may be stored in association with document metadata. Documentmetadata may include training labels for training certain machinelearning models and may additionally include various informationdescribing documents or clauses of documents (e.g., clause metadata),such as user-provided classifications for clauses. In some embodiments,the document management system 140 may analyze or otherwise interpretthe set of documents for analysis 110 in isolation or with reference toa particular subset of documents of the document corpus 130. Thedocument corpus 130 may include documents provided by one or moreparties, such as a party associated with the client device 120. Invarious embodiments, the document corpus 130 may be stored locally onthe document management system 140 or may be stored remotely from thedocument management system 140 (as shown in FIG. 1 ).

The document management system 140 is a computer system (or group ofcomputer systems) for storing and managing documents for variousentities. The document management system 140 receives information fromcomponents of the system environment including the documents foranalysis 110 and the document corpus 130. The document management system140 uses this information to determine information related to thedocuments for analysis, such as deadlines associated with upcomingagreement re-negotiations, updates to clauses in similar documents, andpredictions of certain actions that a client may want to take withrespect to a document. In some cases, the document management system 140may coordinate a collaboration process between a party and acounterparty, for example, between clients associated with client device120A and clients associated with client device 120B.

The document management system 140 can be a server, server group orcluster (including remote servers), or another suitable computing deviceor system of devices. In some implementations, the document managementsystem 140 can communicate with user devices (e.g., the client device120) over the network 180 to receive and provide information (e.g.,documents or document analyses).

The network 180 transmits data within the system environment. Thenetwork 180 may be a local area or wide area network using wireless orwired communication systems, such as the Internet. In some embodiments,the network 180 transmits data over a single connection (e.g., a datacomponent of a cellular signal, or Wi-Fi, among others), or overmultiple connections. The network 180 may include encryptioncapabilities to ensure the security of customer data. For example,encryption technologies may include secure sockets layers (SSL),transport layer security (TLS), virtual private networks (VPNs), andInternet Protocol security (IPsec), among others.

FIG. 2 is a high-level block diagram of a system architecture of adocument management system, in accordance with an example embodiment.The document management system 140 includes various modules and datastores to determine upcoming agreement re-negotiation deadlines, suggestclause updates to related or similar documents, and to suggest actions aclient may want to take with respect to a document. The documentmanagement system 140 includes a database 205, a model generator 210, amodel store 215, a document generator 220, a document review module 225,an expiration identifier, a time prediction module 235, a collaborationmodule 240, an edit detector 245, a document comparison module 250, adocument type predictor 255, an action store 260, and an actionidentifier 265. In FIG. 2 , the modules have been grouped according tosystem processes that they support for clarity. The expirationidentifier 230, the time prediction module 235, and the collaborationmodule 240 are all used by the document management system 140 forpredicting re-negotiation intervals. The edit detector 245 and thedocument comparison module 250 are used by the document managementsystem 140 for updating similar clauses across documents. The documenttype predictor 255, the action store 260, and the action identifier 265are all used by the document management system 140 for suggestingdocument actions to a client. Computer components such as web servers,network interfaces, security functions, load balancers, failoverservers, management and network operations consoles, and the like maynot be shown so as to not obscure the details of the systemarchitecture. Additionally, the document management system 140 maycontain more, fewer, or different components than those shown in FIG. 2and the functionality of the components as described herein may bedistributed differently from the description herein.

The database 205 stores information relevant to the document managementsystem 140. The stored data may include, but is not limited to, the setof documents for analysis 110, clauses within the set of documents foranalysis 110, the document corpus 130, clusters of document clauses, andso on. In some embodiments, the database 205 stores metadata informationassociated with documents or clauses, such as documents labeled withtraining data for machine learning models. The document managementsystem 140 can update information stored in database 205 as newinformation is received, such as new documents for analysis, results ofanalyses performed by the expiration identifier 225, the edit detector230, the time prediction module 235, the document type predictor 240,and the document comparison module 245. The document management system140 can also update information stored in the database 205 based on userinput provided via user interfaces as may be generated by the documentreview module 260.

The model generator 210 trains machine learning models that are used byvarious modules of the document management system 140. In variousembodiments, the model generator 210 may use different versions ofsupervised or unsupervised training, or another training technique togenerate and update the models. The models may be any appropriatemachine learning model, including, but not limited to, decision trees,regression algorithms, support vector machines, and neural networks. Twomodels that may be generated and maintained by the model generator 210include a negotiation prediction model 325 and a document typeprediction model 825. Additional details about these specific models areprovided in reference to FIG. 3 and FIG. 8 . To train the models, themodel generator 210 may access training data, such as labeled documentsstored in the database 205. For example, training data for models maycome from the document corpus 130. The model generator 210 trains themodels using the training data and stores trained machine learningmodels in the model store 215. In some embodiments, the model generatorretrains the models stored in the model store 215 periodically, or asnew training data is received.

The model store 215 stores machine learning models for the documentmanagement system 140. In some embodiments, the model store 215 maystore various versions of models as they are updated over time. In theexample herein, the model store 215 stores a negotiation predictionmodel 325 and a document type prediction model 825. In some cases, themodel store 215 may store multiple versions of a type of model, forexample, to apply to different document types or to other variations ofavailable inputs.

The document generator 220 manages generation and modification ofdocuments. In some cases, a document may be generated at a client device120 and then sent to the document management system 140 for storage oranalysis. In some cases, a document may originate at the documentgenerator 220. The document generator can open new documents forviewing, editing, or other document interactions. Documents may bestarted based on templates stored in the database 205 or based on priordocuments stored in the database 205. In some embodiments, blankdocuments are also generated when needed. In addition to generating newdocuments, the document generator 220 facilitates document edits andupdates. Documents may be edited in various ways. In one embodiment,updated documents are received from a client device 120, and may besaved to the database 205 by the document generator 220 as a new versionof the document. In another embodiment, a user of a client device 120can access an interface of the document management system 140 and canmake edits to a document via the interface. The document generator saveschanges to the database 205. In some cases, the document generator 220can also edit documents without direct input from a client device 120,such as by making updates to documents based on client instructions toupdate similar clauses across all documents, or such as by performing ascheduled document edit. In some embodiments, the document generator 220also manages document deletion.

The document review module 225 facilitates human review of documents anddocument information. In various embodiments, the document review module160 provides one or more user interfaces to client systems associatedwith the document management system 140 for reviewing documents andanalysis of documents. For example, the document review module 225 maytransmit user interfaces for rendering by a client device 120 to presenta document or information about documents to a user of the client device120. Document information presented in the user interfaces may includedocument metadata, predictions about negotiations associated with adocument, or actions that a user may want to take with respect to adocument. The document review module 225 may further use user inputreceived via the provided user interfaces to make changes to one or moredocuments stored in the document store 205. Examples of user interfacesthat may be provided by the document review module 160 are described ingreater detail below with reference to FIG. 5 , FIG. 7 , and FIG. 9 .

Predicting Re-Negotiation Intervals

The document management system 140 can assist users in identifyingupcoming expiring agreements. This can be helpful for example if aclient has multiple agreements that will require renewal negotiations inthe near future. An agreement document can include one or more of acontract, an employment agreement, a purchase agreement, a servicesagreement, or a financial agreement. The document management system 140uses a trained negotiation prediction model to predict how long it islikely to take for agreements to be re-negotiated. Based on theprediction, the document management system 140 may prompt a user of aclient device 120 to begin the renegotiation process at least apredicted interval of time ahead of the expiration date of theagreement. Although examples herein describe agreement re-negotiations,this process may be used in other situations, such as for documents withassociated filing deadlines, or for documents which are regularlygenerated and updated or that are time-sensitive, such as monthlycalendars, or bi-monthly supply request forms. Referring back to FIG. 2, a system for predicting re-negotiation intervals includes theexpiration identifier 230, the time prediction module 235, and thecollaboration module 240.

The expiration identifier 230 tracks information about documents withexpiration dates. Documents stored in the database 205 can be stored inassociation with metadata that indicates expiration dates. In oneembodiment, an expiration date associated with an agreement document isa date after which the terms of the agreement document are no longervalid. For example, a document that represents an agreement between aparty and a counterparty may include a date after which the agreementwill no longer be in force. In some embodiments, an expiration date maybe explicitly indicated in the document metadata when the document issaved to the database 205. In alternate embodiments, the expirationidentifier 230 may use a trained machine learning model or anothertechnique to identify expiration dates associated with documents. In oneembodiment, the expiration identifier determines that all documents thathave a likelihood of being associated with an expiration date haveassociated expiration date metadata stored. For example, the expirationidentifier 230 in some cases may detect an agreement document withoutexpiration date metadata and may request additional information aboutwhether the document has an expiration date from a user of the clientdevice associated with the document. In some embodiments, the expirationidentifier may periodically review the dates associated with storeddocuments and may flag the documents as having upcoming expirationdates.

The time prediction module 235 applies trained machine learning modelsto predict negotiation times for documents with expiration dates. In oneembodiment, the time prediction module 235 uses a negotiation predictionmodel 325, which takes an agreement document as input and outputs apredicted amount of time that it will take for the agreement to bere-negotiated. Additional information about the training and applicationof the negotiation prediction model 325 is included in reference to FIG.3 below. The prediction produced by the time prediction module 235 maybe stored in the database 205 as metadata associated with the agreementdocument. In some cases, the time prediction module 235 may analyze adocument upon receipt of the document at the document management system140. In some embodiments, the time prediction module 235 may update timepredictions for documents periodically, or as additional training datais received to update the negotiation prediction model 325. The documentreview module 225 can provide a notification of upcoming documentexpiration to a user at a client device 120. Using the information aboutthe predicted amount of time that an agreement re-negotiation will takefor an agreement document, the document review module 225 can providethe notifications about upcoming document expirations such that a userof the client device 120 has enough time ahead of the expiration date tore-negotiate the agreement before it expires.

The collaboration module 240 facilitates a collaboration process betweena first party associated with an agreement document and one or morecounterparties to the agreement. In various embodiments, thecollaboration module 240 works in conjunction with the documentgenerator 220 and the document review module 225 to generate a newagreement document for updating and re-negotiating the agreement and tosend notifications and interfaces to the client devices 120 associatedwith the parties and counterparties. The collaboration module 240 maytrack the progress of the re-negotiation of the agreement and may modifyaccounts of a user and any counterparties to enable access by the usersand counterparties to updated agreement documents. For example, theclient device 120B, shown in FIG. 1 may be a client device associatedwith a counterparty to an agreement in a document owned by the clientassociated with client device 120A. When the collaboration module 240begins the re-negotiation process, the document generator 220 mayproduce a new agreement document for review by the client at clientdevice 120A. When the document is ready, the document review module 225may send a notification to the counterparty at client device 120B torequest a review of the document and any additional information orsignatures. In some cases, the collaboration module 240 may keep trackof what additional information is needed to complete an agreementrenegotiation by the expiration deadline. When an agreementre-negotiation is completed by the parties and counterparties, thecollaboration module 240 stores the new agreement documents andassociated expiration dates in the database 205.

FIG. 3 is a flow diagram illustrating training and application processesfor a negotiation prediction model 325, in accordance with an exampleembodiment. The negotiation prediction model 325 takes, as input,metadata and other information related to a document to determine apredicted time it will take to negotiate an agreement document. Thenegotiation prediction model 325 is trained to predict how long anegotiation of an agreement is likely to take based on historicalinformation about the parties involved in the agreement, the type ofagreement, and the contents of the document.

The model generator 210 trains the negotiation prediction model 325using a set of training data 300. In one embodiment, the model generator210 accesses a set of historical agreement documents used for training,determines how long each of the historical agreement documents took tonegotiate, and trains the negotiation prediction model 325 to correlatea length of negotiation for each historical agreement document with atype of each historical agreement document. The training data 300includes historical agreement documents 305 (i.e., past agreementdocuments that have been processed by the document management system 140and associated metadata), historical agreement document types 310 (e.g.,sales contract, indemnity agreement, equipment lease, employmentagreement, non-disclosure agreement, etc.), and historical agreementnegotiation times 315 (e.g., one week, two months, twenty days, etc.).In various embodiments, the training data may include additional ordifferent training inputs, not shown in FIG. 3 . For example, additionaltraining data 300 may include context information about when anagreement was decided and information about the parties involved in pastagreements.

The historical agreement documents 305, historical agreement documenttypes 310, and historical agreement negotiation times 315 may beprovided via client devices 120 to the document management system 140.In other embodiments, the document management system 140 mayautomatically collect the historical agreement document 305, thehistorical agreement document types 310, and the historical agreementnegotiation times 315 to add to the training data 300. In otherembodiments, a user of a client device 120 associated with the documentmanagement system 140 (e.g., a system administrator) may manually inputor curate a subset of the training data 300. It should be noted thatalthough FIG. 3 shows a use of historical documents and historicaldocument information, the training data 300 may also include sampledocuments generated and labeled for the purpose of training thenegotiation prediction model 325 in addition to historical documentdata.

Historical agreement documents 305 used as a subset of the training data300 may include the text of historical documents and metadata associatedwith the documents. Historical agreement documents 305 may includedocuments associated with the user of the client device 120, documentsassociated with an entity or with another user associated with the userof the client device 120, and historical agreement documents may includedocuments associated with users having one or more characteristics incommon with the user. In addition to the text and input fields in adocument, the metadata stored in association with the historicalagreement documents 305 may include document file type, file size,languages within the document, region in which the document originated,characteristics associated with the sending and receiving party of thedocument (e.g., size, industry, location of headquarters, revenue,corporate structure), types or categories of information or passageswithin the document, and the like.

The historical agreement document types 310 include annotations of thehistorical agreement document 305 that indicate the type of agreement ornegotiation that the document represents. An agreement document type mayinclude any document with an agreement negotiated between at least twoparties, such as a sales contract, a permission slip, a rental or leaseagreement, a liability waiver, a financial document, an investment termsheet, a purchase order, an employment agreement, a mortgageapplication, an indemnity agreement, an equipment lease, and anon-disclosure agreement.

The historical agreement negotiation times 315 include annotations ofthe historical agreement documents 305 that indicate the length of timeit took for the agreement represented by the document to be reached. Insome cases, the training information about the historical agreementnegotiation times may be input manually by a user of a client device 120associated with the document or by a system administrator. In somecases, the document management system 140 may detect an amount of timeit took for an agreement to be negotiated (e.g., the time betweendocument creation and execution of the document by all parties).

The model generator 210 uses supervised or unsupervised machine learningto train the negotiation prediction models 325 using the training data300. Different machine learning techniques may be used in variousembodiments, such as linear support vector machines (linear SVM),boosting for other algorithms (e.g., AdaBoost), neural networks,logistic regression, naive Bayes, memory-based learning, random forests,bagged trees, decision trees, boosted trees, or boosted stumps. Thetraining of the negotiation prediction model 325 helps the model toidentify relationships between the historical agreement documents 305,the historical agreement document types 310, and the historicalnegotiation times 315. In other words, training a negotiation predictionmodel 325 enables the negotiation prediction model 325 to identify apredicted time 330 that it will take for an agreement to bere-negotiated, given the original agreement document 320 and history ofsimilar agreement documents. In some embodiments, the model generator210 trains multiple negotiation prediction models 325, such as aseparate negotiation prediction model 325 for each agreement type.

A trained negotiation prediction model 325 can be applied by the timeprediction module 235 to an agreement document 320. In some embodiments,the negotiation prediction model 325 also accepts metadata associatedwith the document (e.g., document agreement type, agreement parties) asinput. The negotiation prediction model 325 generates a predicted time330 that it will take for the agreement in the current agreementdocument to be renegotiated. The document review module 225 may displaythe agreement expiration date from the expiration identifier 230 and thepredicted time 330 for re-negotiation of the agreement for a clientassociated with the document to view in an interface at the clientdevice 120.

FIG. 4 illustrates an interface 400 of a document analytics dashboardfor reviewing upcoming expiration dates of agreement documents, inaccordance with an example embodiment. In the embodiment shown, theinterface 400 includes a document list 410, notification icons 420, anagreement status interface 430, and a re-negotiation widget 440. Theexample interface 400 shows a document list 410 of agreement documents.In the example of FIG. 4 , a user may also search for specific documentsin the list using a search interface. The document list includesnotification icons 420 next to agreements with identified expirationdates that are sooner than a threshold length of time from the presentdate. For example, AgreementB.docx, AgreementF.docx, and AgreementG.docxhave notification icons 420 indicating impeding expiration dates. Theinterface 400 also includes an agreement status interface 430 thatdisplays additional details about the expiration dates of agreementdocuments in the document list 410. In the example of FIG. 4 ,AgreementF.docx is selected by the user, as indicated by the dotted fillin the document list 410. The agreement status interface 430 displays anexpiration date of the document to be Feb. 3, 2022, as identified by theexpiration identifier 230. The agreement status interface 430 alsodisplays an estimated time for negotiating a new agreement to be fourweeks, as generated by the time prediction module 235. In the example ofFIG. 4 , the interface 400 also displays a document preview so that auser can view the selected document. The re-negotiation widget 440 maybe a button or other interface configuration for confirming a selection.The user can select the re-negotiation widget 440 to confirm a requestto generate an updated version of the agreement document and to initiatea document collaboration process between the user and a counterparty tothe agreement, as may be managed by the collaboration module 240.Initiating the document collaboration process may include modifyingaccounts of the user and the counterparty to enable access to theupdated agreement document. The components and functionalities of aninterface 400 may be different in different embodiments.

FIG. 5 illustrates an example process for identifying upcoming expiringagreement documents and predicting a length of time for re-negotiatingthe agreement, in accordance with an embodiment. In the embodiment shownin FIG. 5 , the process is performed by the document management system140. In other embodiments, some or all of the steps of the process maybe performed by other components of the system environment or may beperformed in a different order than that depicted in FIG. 5 .Additionally, in other embodiments, the process illustrated in FIG. 5can include fewer, additional, or different steps than those describedherein.

The document management system 140 identifies 510 an expiration dateassociated with each of a set of agreement documents corresponding to auser. For example, the expiration identifier 230 may identify anexpiration date associated with each of a user's agreement documentsstored in the data store 205. The time prediction module 235 of thedocument management system 140 applies 520 a machine learning model,such as the negotiation prediction model 325, to a first agreementdocument of the set of agreement documents. The machine learning modelis trained on historical agreement documents associated with a samedocument type as the first agreement document. The model is configuredto output a predicted time of negotiation for the first agreementdocument. At a time more than the predicted time of negotiation for thefirst agreement document before the expiration date, the documentmanagement system 140 populates 530 a notification interface presentedto the user, e.g., at a client device 120, with a reminder notificationidentifying an expiration date of the first agreement document and thepredicted time of negotiation for the first agreement document. Inresponse to a confirmation from the user via the displayed notification,the document generator 220 of the document management system 140generates 540 an updated first agreement document and the collaborationmodule 240 of the document management system 140 initiates 540 adocument collaboration process between the user and a counterparty. Forexample, the document may be sent to the counterparty for review andexecution.

Clause Updates Based on Edits to Similar Clauses

The document management system 140 monitors standard clauses included bya user in documents and, when a clause is updated, flags other documentsthat include the clause. In this way, a user can be notified of otherdocuments that the user may also want to update to include the languageof the updated clause. If the difference between the instance of eachclause in a flagged document and the updated clause language exceeds athreshold value, the documents may be presented to the user within aninterface that enables the user to automate the process of updating thelanguage of the outdated clause instances in the documents. Referringback to FIG. 2 , a system for updating similar clauses includes the editdetector 245 and the document comparison module 250.

The edit detector 245 monitors changes made to documents to detect whenclauses have been edited. In various embodiments, the edit detector 245may detect changes to edits in different ways. In some cases, thedocument management system 140 may receive updated documents from aclient device 120, such as documents for analysis 110. The edit detector245 may compare a new version of a document received from the clientdevice 120 with the same document already stored on the database 205 todetermine if any edits have been made to clauses in the document. In oneembodiment, a user of a client device 120 can make edits to a documentby interacting directly with a document editing interface sent to theclient device by the document review module 225. The edit detector 245my detect when a change is made to the language of a clause through thedocument editing interface. In some embodiments, identifying an editmade by a user to the language of a clause includes detection of an editmade by the user to language to a clause within an agreement documentthat is being negotiated by the user during a negotiation process. Anedited clause may include, but is not limited to, a name change of anentity listed in the clause, a change to a quantity or numerical amountwithin the clause, a change of jurisdiction or geographic locationdescribed in the clause, a change to a liability within the clause, achange to a date within the clause, a change to a proper noun within theclause, or a change to legal terminology within the clause.

The document comparison module 250 identifies a subset of documentsstored in the database 205 that include an outdated instance of theupdated clause. To identify the related documents, the documentcomparison module 250 queries the corpus of documents stored in thedatabase 205 that are associated with the user. From this corpus ofdocuments, the document comparison module 250 reviews the contents ofthe documents and identifies the related documents having the originalinstance of the clause (i.e., as it was before it was edited). Thedocument comparison module 250 compares the edited clause with theoriginal instance of the clause in the identified related documents anddetermines whether to include each of the related documents in a subsetof documents that will be suggested to a user for editing to include theupdated clause (i.e., a subset of the documents that include an outdatedinstance of the clause). In one embodiment, to determine the subset ofdocuments, the document comparison module 250 identifies the subset ofdocuments that include instances of the original clause that differ fromthe edited version of the clause by more than a threshold amount. As oneexample, a threshold of a difference of at least 10 characters may berequired between the edited clause and the version of the originalclause in another document for the other document to be included in thesubset of related documents. Once the subset of documents is determinedby the document comparison module 250, the document review module 225can modify a document interface presented to the user at a client device120 to include an interface element that identifies the edited clauseand that also displays the identified subset of document that includethe original, now outdated, clause. The user may interact with thedocument management system 140 via the interface presented by thedocument review module 225 to select one or more of the identifieddocuments for replacement of the outdated clause with an instance of theupdated clause, and the selected documents may be updated by thedocument generator 220.

FIG. 6 illustrates an interface 600 of a document editing dashboard forreviewing similar clauses across separate documents, in accordance withan example embodiment. In the embodiment shown, the interface 600includes a document list 610, a first interface area 620, a secondinterface area 630, markup indications 640, and a confirmation widget650. The example interface 600 shows a document list 610 of documentsthat the document comparison module 250 identified as a subset of thedocuments associated with the user that include a clause that matches ornearly matches an edited clause in a document of the user. The exampleinterface also includes a first interface area 620 that displays theedited clause and a second interface area 630 that displays a version ofthe original outdated clause in another document. In variousembodiments, the first interface area 620 and/or the second interfacearea 630 includes markup 640 or another indication (e.g., highlighting)of the differences between the edited clause and the outdated originalclause. In the example of FIG. 6 , the first interface area 620 showstext from clause 10.1 of a document called Contract1.docx. The documentlist 610 lists documents that the document comparison module 250 hasidentified as having the same or a similar original clause (e.g., clause10.1). In the example, document list 610, document AgreementC.docx isselected, as indicated by the dotted fill. Consequently, text from therelated original clause in AgreementC.docx is displayed in the secondinterface area 630. The markup 640 in the first interface area 620 showsthe edits that were made to the clause in Contract1.docx that make itdiffer from the original clause language which is still present inAgreementC.docx. In the example interface 600 of FIG. 6 , a user canselect among the documents in the document list 610 using checkmarks incheck boxes to indicate one or more documents should be modified by thedocument generator 220 to include the updated clause language. Toconfirm the documents for modification, the user can select theconfirmation widget 650. The components and functionalities of aninterface 600 may be different in different embodiments.

FIG. 7 illustrates an example process for flagging clauses in documentsfor updating when a similar clause is updated in another document, inaccordance with an embodiment. In the embodiment shown in FIG. 7 , theprocess is performed by the document management system 140. In otherembodiments, some or all of the steps of the process may be performed byother components of the system environment or may be performed in adifferent order than that depicted in FIG. 7 . Additionally, in otherembodiments, the process illustrated in FIG. 7 can include fewer,additional, or different steps than those described herein.

When a user edits a document, the edit detector 245 of the documentmanagement system 140 identifies 710 an edit made by a user to languageof a first clause associated with the user. The document comparisonmodule 250 of the document management system 140 queries 720 a corpus ofdocuments associated with the user to identify a set of documents thatinclude an instance of the first clause. The document comparison module250 compares 730 the edited first clause with the instance of the firstclause in each of the set of documents to identify a subset of the setof documents that include an outdated instance of the first clause.Based on the identified subset of documents, the document review module225 modifies 740 a document interface presented to the user to includean interface element that identifies the edited first clause and thesubset of documents that include an outdated instance of the firstclause. The user may interact with the modified interface to view thedifferences between the document clauses. In response to a confirmationinteraction by the user via the interface element, the documentgenerator 220 of the document management system 140 modifies 750 (e.g.,edits) each of one or more of the subset of documents by replacing theoutdated instance of the first clause in the document with the editedfirst clause. For example, each of the subset of documents that the userselected may be edited to include the updated clause language.

Machine-Learned Document Action Suggestions

The document management system 140 uses machine learning models toidentify a type of document with which a user is interacting and tosuggest actions that can be taken on the document. Examples of actionsthat may be taken with respect to a document include replacing text withfields, replacing clauses with pre-approved versions of the clauses, andsynchronizing the document with a third-party system. Action options maybe based on actions taken by similar users, actions done to similardocuments, and actions taken in similar contexts (e.g., at similar timesof the year). In one embodiment, the document management system 140presents the recommended actions within an interface that enables a userto view and select desired actions from a list of suggested actions andthat enables the user to automate the performance of the actions ifrequested. Referring back to FIG. 2 , a system for suggesting documentactions includes the document type predictor 255, the action store 260,and the action identifier 265.

The document type predictor 255 applies trained machine learning modelsto predict a type of document that a user is editing or otherwiseinteracting with. In one embodiment, the document type predictor 255uses a document type prediction model 825, which takes in a document, apartial document, and/or a set of document features as input and outputsa predicted type of the document. The document type predictor 255 mayanalyze a document to determine a set of associated document features.In some embodiments, information about features associated with adocument may additionally or alternately be stored as metadata inrelation to the document at the database 205. Document featuresidentified by the document management system 140 may include terms usedwithin the document, clauses used within the document, images within thedocument, entities associated with the document, permissions associatedwith the document, actions taken on the document, templates used togenerate the document, characteristics of the user, and characteristicsof entities associated with a document. Additional information about thetraining and application of the document type prediction model 8125 isincluded in reference to FIG. 8 below. The document type predictiongenerated by the document type predictor 255 may be stored in thedatabase 205 as metadata associated with the document. In some cases,the document type predictor 255 may analyze a document upon receipt ofthe document at the document management system 140 or the document typepredictor 255 may update document type predictions by reanalyzingdocuments stored in the database 205 periodically or as additionaltraining data is received to update the document type prediction model825. In some embodiments, the document type predictor 255 analyzes adocument or partial document to determine the type when the user isactively interacting with the document, for example, via an editing orreview interface provided by the document review module 225. Thedocument type prediction generated by the document type predictor 255 isprovided to the action identifier for use in determining appropriateactions to suggest to a user with respect to the document.

The action store 260 stores information about actions that the documentmanagement system 140 can perform on documents. This may includepossible actions that can be performed, as included, for example, in theaction store by a system administrator or other user. Examples ofactions that can be performed for a document include replacing text withfields, adding signature fields, replacing text with pre-approvedversions of clauses, synchronizing the document with an externaldocument system, changing a tense used in some or all sections of thedocument, populating fields with data from external data sources, andproviding the document for review or signature to an entity associatedwith the document. The action store 260 may additionally store relatedinformation and metadata about possible actions. For example, the actionstore 260 may store information about which actions are performed ondifferent document types, and how often certain actions are performed ondifferent document types. Other metadata stored by the action store 260could include common combinations of actions that have been taken byusers on documents, a number of users that have performed certain actiontypes, a number of times an action has been taken on documents, andcharacteristics of users that take perform certain actions with respectto a document type.

The action identifier 265 identifies a set of actions that can be takenon a document that user is currently editing or reviewing. The set ofactions can then be suggested to the user via an interface presented ata client device 120 as generated by the document review module 225. Theaction identifier 265 identifies a set of actions that can be taken onthe document by accessing the document type of the current document aspredicted by the document type predictor 255. The action identifier 265then accesses the action store 260 to obtain a set of actions that havebeen taken or can be taken on documents of the same type. In someembodiments, actions taken on other documents of the document typeinclude actions taken by the same user on other documents of thedocument type or actions taken by users with one or more characteristicsin common with the user. In some embodiments, actions taken on otherdocuments of the document type include actions taken in the past by atleast a threshold number of users or actions that have been implementedat least a threshold number of times on other documents of the documenttype.

FIG. 8 is a flow diagram illustrating training and application processesfor a document type prediction model 825, in accordance with an exampleembodiment. The document type prediction model 825 takes, as input, adocument 820 (or a partial document or list of document featuresassociated with the document 820), and outputs a predicted type of thedocument. The document type prediction model 825 is trained to predict atype of a document using information about sample documents, documentfeatures and document types.

The model generator 210 trains the document type prediction model 825using a set of training data 800. In some cases, multiple document typeprediction models 825 may be trained and stored in the model store 215for use by the document type predictor 255. For example, the model store215 may store a separate document type prediction model 825 inassociation with each individual user or in association with separateentities using the document management system 140. The document typeprediction model 825 is trained to identify correlations betweendocument types and document features and the document type of a documentmay be determined based on an input set of document features using theidentified correlations. The training data 800 may include sampledocuments 805, sample document features 810, and sample document types815. In various embodiments, the training data 800 may includeadditional or different training inputs not shown in FIG. 8 .

The training data 800 may be provided via client device 120 to thedocument management system 140. In other embodiments, the documentmanagement system 140 may automatically collect the sample documents805, the sample document features 810, and the sample document types 815to add to the training data 800 as it processes documents. In otherembodiments, a user of a client device 120 associated with the documentmanagement system 140 (e.g., a system administrator) may manually inputor curate a subset of the training data 800. It should be noted that thetraining data 800 may include historical document data and/or sampledocuments that are generated and labeled for the purpose of training thedocument type prediction model 825.

Sample documents 805 used as a subset of the training data 800 mayinclude the text of documents and metadata associated with thedocuments. The training set of sample documents 805 may includedocuments associated with a particular user, documents associated withan entity or with another user associated with the user, or documentsassociated with users having one or more characteristics in common withthe user. For example, if the model generator 210 trains a model foreach user, then it may use documents associated with similar users fortraining. In addition to the text and input fields present in adocument, the metadata stored in association with a sample document mayinclude document file type, file size, languages within the document,region in which the document originated, characteristics associated withthe sending and receiving party of the document (e.g., size, industry,location of headquarters, revenue, corporate structure), types orcategories of information or passages within the document, and the like.

The sample document features 810 used for training the machine learningmodels may include descriptions of document features or may includeannotations of sample documents 805 as having certain document features.Document features 810 may include specific terms used within a document,clauses within a document, images within a document, entities associatedwith a document, permissions associated with the document, actions takenon a document in the past, templates used to generate a document,characteristics of a user associated with the document, characteristicsof entities associated with a document, a word count, the presence ofsignature lines, and specific data collection fields included in thedocument, among other items.

The sample document types 815 include annotations to the sampledocuments 805 and to the sample document features 810 that indicate thetype of document with which the documents or features can be associated.For example, this may include a document type label associated with asample document 805. In some cases, training information about documenttypes may be input manually by a user of a client device 120 associatedwith the document or by a system administrator.

The model generator 210 uses supervised or unsupervised machine learningto train the document type prediction models 825 using the training data800. Different machine learning techniques may be used in variousembodiments, such as linear support vector machines (linear SVM),boosting for other algorithms (e.g., AdaBoost), neural networks,logistic regression, naïve Bayes, memory-based learning, random forests,bagged trees, decision trees, boosted trees, or boosted stumps. Thetraining of a document type prediction model 825 helps the model toidentify correlations between document features and document types. Thatis, training a document type prediction model 825 enables the documenttype prediction model 825 to identify a predicted type 830 of a document820. In some embodiments, the model generator 210 trains multipledocument type prediction models 825, such as a separate document typeprediction model 825 for each user or for each entity associated withthe document management system.

A trained document type prediction model 825 can be applied by thedocument type predictor 255 to a document 820. In some embodiments, thedocument type prediction model 825 also or alternately accepts metadataassociated with the document (e.g., document features) as input. Thedocument type prediction model 825 generates a predicted type 830 of thedocument 820. The action identifier 265 can use a predicted documenttype 830 to select actions that are likely to be relevant to thedocument 820 from the action store 260. The document review module 225may display the selected actions for a user of a client device 120 toview and select so that the action can be taken for the document.

FIG. 9 illustrates an interface 900 of a document editing dashboard forviewing and selecting suggested actions related to a document, inaccordance with an example embodiment. In the embodiment shown, theinterface 900 includes an action list 910, and action confirmationwidgets 920. The example interface 900 also displays a view of thedocument (e.g., Contract1.docx) that the user is currently reviewing orediting. The action list 910 has been modified by the document reviewmodule 225 to present actions selected by the action identifier 265 asbeing available actions to perform on the document. The user can confirmwhich, if any, of the actions for the document management system 140 totake with respect to the document by selecting one or more of the actionconfirmation widgets 920. An action confirmation widget 920 may be abutton, a drop-down menu, an input field, or another user inputcomponent of the interface 900. In response to a user selecting one ormore of the actions via the interface 900, the document managementsystem 140 performs the selected actions on the document.

FIG. 10 illustrates an example process for identifying actions that auser can request for a document, in accordance with an embodiment. Inthe embodiment shown in FIG. 10 , the process is performed by thedocument management system 140. In other embodiments, some or all of thesteps of the process may be performed by other components of the systemenvironment, or may be performed in a different order than that depictedin FIG. 10 . Additionally, in other embodiments, the process illustratedin FIG. 10 can include fewer, additional, or different steps than thosedescribed herein. When a user is interacts with a document, for exampleto edit or review the document, the document management system 140 cansuggest actions related to the document. The document type predictor 255of the document management system 140 applies 1020 a machine-learningmodel to a set of features of the document. The machine learning modelmay be a document type prediction model 825 that is trained on atraining set of documents. Each of the documents in the training set ofdocuments may include information about training document features andmay be tagged or labeled with a training document type. The trainedmachine learning model is configured to output a document type of thedocument based on the set of features associated with the givendocument. Based on the document type output by the machine learningmodel for the document, the action identifier 265 of the documentmanagement system 140 accesses an action store 260 and identifies 1030 aset of actions that can be taken on the document based on the contentsof the document and actions taken on other documents of the samepredicted document type. The document review module 225 of the documentmanagement system 140 modifies 1040 a document interface presented tothe user at a client device 120 to include an interface element thatidentifies the set of actions that can be taken on the document. Theuser may then choose to confirm via the interface that the displayedactions should be taken on the document. In response to a selection ofone or more of the set of actions via the interface element by the user,the document management system 140 performs 1050 the selected actions onthe document.

Additional Configuration Considerations

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

Some portions of this description describe the embodiments in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are commonly used bythose skilled in the data processing arts to convey the substance oftheir work effectively to others skilled in the art. These operations,while described functionally, computationally, or logically, areunderstood to be implemented by computer programs or equivalentelectrical circuits, microcode, or the like.

Furthermore, it has also proven convenient at times, to refer to thesearrangements of operations as modules, without loss of generality. Thedescribed operations and their associated modules may be embodied insoftware, firmware, hardware, or any combinations thereof

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computing deviceselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a non-transitory,tangible computer readable storage medium, or any type of media suitablefor storing electronic instructions, which may be coupled to a computersystem bus. Furthermore, any computing systems referred to in thespecification may include a single processor or may be architecturesemploying multiple processor designs for increased computing capability.

Embodiments may also relate to a product that is produced by a computingprocess described herein. Such a product may comprise informationresulting from a computing process, where the information is stored on anon-transitory, tangible computer readable storage medium and mayinclude any embodiment of a computer program product or other datacombination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the patent rights. It istherefore intended that the scope of the patent rights be limited not bythis detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thepatent rights, which is set forth in the following claims.

What is claimed is:
 1. A method comprising: identifying, by a documentmanagement system, an edit made by a user to language of a first clauseassociated with the user; querying, by the document management system, acorpus of documents associated with the user to identify a set ofdocuments that include an instance of the first clause; comparing, bythe document management system, the edited first clause with theinstance of the first clause in each of the set of documents to identifya subset of the set of documents that include an outdated instance ofthe first clause; modifying, by the document management system, adocument interface presented to the user to include 1) a first interfaceportion listing the identified subset of documents and configured toenable the user to select a document of the identified set of documents,2) a second interface portion that displays the edited first clause, and3) a third interface portion that displays an outdated instance of thefirst clause within a selected document that is selected from theidentified subset of documents using the first interface portion; and inresponse to a confirmation interaction by the user via the interfaceelement, modifying, by the document management system, each of one ormore of the subset of documents by replacing the outdated instance ofthe first clause in the document with the edited first clause.
 2. Themethod of claim 1, wherein identifying an edit made by the user to thelanguage of the first clause comprises detecting an edit made by theuser to language within an agreement document being negotiated by theuser corresponding to the first clause during a negotiation process. 3.The method of claim 1, wherein the identifying subset of documentscomprise documents that include instances of the first clause thatdiffer from the edited first clause by more than a threshold amount. 4.The method of claim 1, wherein the edited first clause includes a changeto a name of an entity within the first clause, a change to a quantityor numerical amount within the first clause, a change of jurisdiction orgeographic location within the first clause, a change to a liabilitywithin the first clause, a change to a date within the first clause, achange to a proper noun within the first clause, or a change to legalterminology within the first clause.
 5. The method of claim 1, whereinone of the first interface portion or the second interface portionincludes highlighting or markup indicating one or more differencesbetween the edited first clause and the outdated instance of the firstclause.
 6. The method of claim 1, wherein the confirmation interactionby the user comprises a selection of the one or more of the subset ofdocuments to modify.
 7. A non-transitory computer-readable storagemedium storing executable instructions that, when executed by a hardwareprocessor, cause the hardware processor to perform steps comprising:identifying, by a document management system, an edit made by a user tolanguage of a first clause associated with the user; querying, by thedocument management system, a corpus of documents associated with theuser to identify a set of documents that include an instance of thefirst clause; comparing, by the document management system, the editedfirst clause with the instance of the first clause in each of the set ofdocuments to identify a subset of the set of documents that include anoutdated instance of the first clause; modifying, by the documentmanagement system, a document interface presented to the user toinclude 1) a first interface portion listing the identified set ofdocuments and configured to enable the user to select a document of theidentified subset of documents, 2) a second interface portion thatdisplays the edited first clause, and 3) a third interface portion thatdisplays an outdated instance of the first clause within a selecteddocument that is selected from the identified subset of documents usingthe first interface portion; and in response to a confirmationinteraction by the user via the interface element, modifying, by thedocument management system, each of one or more of the subset ofdocuments by replacing the outdated instance of the first clause in thedocument with the edited first clause.
 8. The non-transitorycomputer-readable storage medium of claim 7, wherein identifying an editmade by the user to the language of the first clause comprises detectingan edit made by the user to language within an agreement document beingnegotiated by the user corresponding to the first clause during anegotiation process.
 9. The non-transitory computer-readable storagemedium of claim 7, wherein the identifying subset of documents comprisedocuments that include instances of the first clause that differ fromthe edited first clause by more than a threshold amount.
 10. Thenon-transitory computer-readable storage medium of claim 7, wherein theedited first clause includes a change to a name of an entity within thefirst clause, a change to a quantity or numerical amount within thefirst clause, a change of jurisdiction or geographic location within thefirst clause, a change to a liability within the first clause, a changeto a date within the first clause, a change to a proper noun within thefirst clause, or a change to legal terminology within the first clause.11. The non-transitory computer-readable storage medium of claim 7,wherein one of the first interface portion or the second interfaceportion includes highlighting or markup indicating one or moredifferences between the edited first clause and the outdated instance ofthe first clause.
 12. The non-transitory computer-readable storagemedium of claim 7, wherein the confirmation interaction by the usercomprises a selection of the one or more of the subset of documents tomodify.
 13. A document management system comprising: a hardwareprocessor; and a non-transitory computer-readable storage medium storingexecutable instructions that, when executed, cause the hardwareprocessor to perform steps comprising: identifying, by the documentmanagement system, an edit made by a user to language of a first clauseassociated with the user; querying, by the document management system, acorpus of documents associated with the user to identify a set ofdocuments that include an instance of the first clause; comparing, bythe document management system, the edited first clause with theinstance of the first clause in each of the set of documents to identifya subset of the set of documents that include an outdated instance ofthe first clause; modifying, by the document management system, adocument interface presented to the user to include 1) a first interfaceportion listing the identified subset of documents and configured toenable the user to select a document of the identified subset ofdocuments, 2) a second interface portion that displays the edited firstclause, and 3) a third interface portion that displays an outdatedinstance of the first clause within a selected document that is selectedfrom the identified subset of documents using the first interfaceportion; and in response to a confirmation interaction by the user viathe interface element, modifying, by the document management system,each of one or more of the subset of documents by replacing the outdatedinstance of the first clause in the document with the edited firstclause.
 14. The document management system of claim 13, whereinidentifying an edit made by the user to the language of the first clausecomprises detecting an edit made by the user to language within anagreement document being negotiated by the user corresponding to thefirst clause during a negotiation process.
 15. The document managementsystem of claim 13, wherein the identifying subset of documents comprisedocuments that include instances of the first clause that differ fromthe edited first clause by more than a threshold amount.
 16. Thedocument management system of claim 13, wherein the edited first clauseincludes a change to a name of an entity within the first clause, achange to a quantity or numerical amount within the first clause, achange of jurisdiction or geographic location within the first clause, achange to a liability within the first clause, a change to a date withinthe first clause, a change to a proper noun within the first clause, ora change to legal terminology within the first clause.
 17. The documentmanagement system of claim 13, wherein one of the first interfaceportion or the second interface portion includes highlighting or markupindicating one or more differences between the edited first clause andthe outdated instance of the first clause.